Use nostr as cache system #1362

KoalaSat · 2024-07-01T16:06:48Z

What does this PR do?

This is a first POC. I'm exploring to use nostr not as primary DB but as a cache system of a decentralized network such as the federation of Robosats, right now a client has to directly connect to every single coordinator book to fetch its specific orders.

Centralizing this system would create a single point of failure, so I'm exploring a federation cache network, where the full book is aditioanlly stored and maintained by coordinators, allowing users to initially connect just to 2-3 coordinators. any feedback is welcome 🙂

For now this PR just creates a coordinator relay and sends an event every time a new order is created. The note follows this on going NIP nostr-protocol/nips#1331

As additional win, this will allow Robosats to quickly integrate others' platforms orders with just a single implementation

Federation cache

Every relay of the federation will be in sync with the others, containing all orders available in the federation:

flowchart TD;
    classDef coordinator fill:#F7931A
    classDef nostr fill:#8e30eb
    subgraph 1
    Coor1:::coordinator-->Relay1;
    end
    subgraph 2
    Coor2:::coordinator-->Relay2;
    end
    subgraph 3
    Coor3:::coordinator-->Relay3;
    end

    Relay1:::nostr<-->Relay2;
    Relay2:::nostr<-->Relay3;
    Relay3:::nostr<-->Relay1;
    Relay2<-->ExternalRelay;

    Relay1-->Client;
    Relay2-->Client;

The federation will keep a strong negentropy circle between their relays, but any other external relay will be able to fetch or create new order events. Because it's just cache, we can leave the creation of events open to anyone, if it becomes a spam problem, it can be cleaned up automatically by coordinators.

Clients will always send requests filtering by coordinators' public keys, that way spam or fake orders will never reach to the user.

How to test

Publish an order. Connect your WS tools to ws://localhost:12596/nostr and send:

["REQ", "111", {"kinds":[38383]}]

You should get in return:

["EVENT", "111", <your order note>]
["EOSE","111"]  # End of response

Any change on the order should remove previous note and create a new one (identified over time by its d tag). To keep privacy, Robosats will only have 2 order statuses: pending and success.

Checklist before merging

Install pre-commit and initialize it: pip install pre-commit, then pre-commit install. Pre-commit installs git hooks that automatically check the codebase. If pre-commit fails when you commit your changes, please fix the problems it points out.

jerryfletcher21 · 2024-07-12T11:12:52Z

@KoalaSat nice idea, and really needed once the number of coordinators starts to grow. :)
An other thing that we will have to change related to this improvement is that in the main client when creating a robot, the robot is created in every coordinator, again making a requests to each of them. This is not really needed since the robot will then be used on just one coordinator.

KoalaSat · 2024-07-12T12:58:21Z

@KoalaSat nice idea, and really needed once the number of coordinators starts to grow. :) An other thing that we will have to change related to this improvement is that in the main client when creating a robot, the robot is created in every coordinator, again making a requests to each of them. This is not really needed since the robot will then be used on just one coordinator.

The reason behind is that wherever the token comes from, we ask for coordinators if the token exists, in case there are active or past orders for it. Certainty, we know when the token is created but that would involve to fetch the coordinator every time the user tries to create an order, affecting the waiting time (unknown robots can´t create an order).

It's tricky to find an efficient way and don´t affect other parts, specially because the app/api assumes my robot is known, and for example, we would have to handle it in case the user visits their profile.

Anyways itś good to raise this topic from time to time to figure out if there are better ways

jerryfletcher21 · 2024-07-12T13:38:48Z

In roboauto I have that when creating a robot, the user have to specifiy a coordinator, and when recovering a robot, together with the secret also the coordinator have to be specified.
A similar solution could also be implemented in the main client, especially once the number of coordinators starts to grow.
It can also be added an additional button that searches across the coordinator if the robot with the specified secret is present, to be used just when the user is recovering a robot and does not remember his coordinator.
In this way except when pressing this button the client never makes a requests to every coordinator.

KoalaSat · 2024-07-12T17:48:12Z

In roboauto I have that when creating a robot, the user have to specifiy a coordinator, and when recovering a robot, together with the secret also the coordinator have to be specified. A similar solution could also be implemented in the main client, especially once the number of coordinators starts to grow. It can also be added an additional button that searches across the coordinator if the robot with the specified secret is present, to be used just when the user is recovering a robot and does not remember his coordinator. In this way except when pressing this button the client never makes a requests to every coordinator.

Good to know, my plan with this cache service was also very related to this topic, cache book can also store on goings orders, and if you download it and just search on it your robot name thatś another nice way to know there is an order for your robot, as far the cache system is reliable enought 😄

jerryfletcher21 · 2024-07-13T11:01:23Z

Good to know, my plan with this cache service was also very related to this topic, cache book can also store on goings orders, and if you download it and just search on it your robot name thatś another nice way to know there is an order for your robot, as far the cache system is reliable enought 😄

Ah yes this is even better, if the cache book keeps also track of ongoing orders, the user needs just the secret. The client can calculate the robot name from the secret, then search for his coordinator in the cache book. So there is never the need to search all coordinators to check if the robot is present and have an order.
But I agree is better to first work on making the cache system reliable :)

KoalaSat · 2024-07-20T13:06:15Z

docker/strfry/crontab

+# For more information see the manual pages of crontab(5) and cron(8)
+#
+# m h  dom mon dow   command
+*/1 * * * * torsocks /app/sync.sh


torfication of sync calls are done as part of the cron command to avoid mistakes if the coordinator modifies sync.sh https://github.com/RoboSats/robosats/pull/1362/files#diff-366e6549a96aec09107bfc5f3f3804f92cfee8d9804bc8b222136d0a0a15d08b

KoalaSat · 2024-08-09T09:19:37Z

tested and working in Satstralia 👍

KoalaSat · 2024-09-06T08:18:53Z

Tested in Satstralia for 3 weeks already and fully working! 🚀

KoalaSat force-pushed the use-nostr-as-cache-system branch 2 times, most recently from 8d8e716 to fa90ccc Compare July 8, 2024 12:41

KoalaSat force-pushed the use-nostr-as-cache-system branch 2 times, most recently from db49deb to c4452b7 Compare July 16, 2024 11:18

KoalaSat marked this pull request as ready for review July 19, 2024 10:55

KoalaSat requested a review from Reckless-Satoshi July 19, 2024 10:55

KoalaSat commented Jul 20, 2024

View reviewed changes

KoalaSat force-pushed the use-nostr-as-cache-system branch 2 times, most recently from 9692a7e to 5000155 Compare July 24, 2024 16:36

KoalaSat force-pushed the use-nostr-as-cache-system branch 6 times, most recently from 7cba270 to b658679 Compare August 7, 2024 20:30

KoalaSat and others added 10 commits August 7, 2024 22:43

Use nostr as cache system

3e460bb

Prepare for other events

97bb2dc

Switch to strfry

bdc10eb

NIP corrections

cbb063f

Addapt Docker

6635722

Testing

a6c2130

Better tags

c597cc8

Test entrypoint

c1e9da5

Working

b7f9f18

Sync filtering

334587b

KoalaSat added 16 commits August 7, 2024 22:43

CR

be3bd80

Real onions

9b7240e

Some NIP fixes

c28ba4a

Filter as variable in sync.sh

ae45029

Fix unmutable conf

ae18acd

Fix unmutable conf 2

7d0b3ba

Fix unmutable conf 3

fda906b

Tesnet sync

73189af

Tesnet onions

c0d0872

Docker write access

ab1a488

Logs to docker

507082d

executable sync

e3e530c

executable sync

dd35a3e

Remove test dockers

19fc1f0

Async to sync

af3a03c

Review nostr tags

032a48a

KoalaSat force-pushed the use-nostr-as-cache-system branch from b658679 to 032a48a Compare August 7, 2024 20:49

Fix integer tag

1c7b2a8

KoalaSat force-pushed the use-nostr-as-cache-system branch from 1b02f30 to 1c7b2a8 Compare August 7, 2024 21:34

KoalaSat added 4 commits August 8, 2024 00:19

Fix integer tag

cad1a86

Fix url tag

3433f45

Remove created at

d79f27b

strfry bug free version

7ef8a1b

KoalaSat added 2 commits August 9, 2024 17:06

Better sync urls

58d21d9

Merge branch 'main' into use-nostr-as-cache-system

e9620e3

KoalaSat merged commit ce8799d into main Sep 6, 2024
2 checks passed

KoalaSat deleted the use-nostr-as-cache-system branch September 6, 2024 08:19

KoalaSat mentioned this pull request Nov 8, 2024

Coordinator ratings over Nostr #1097

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use nostr as cache system #1362

Use nostr as cache system #1362

KoalaSat commented Jul 1, 2024 •

edited

Loading

jerryfletcher21 commented Jul 12, 2024 •

edited

Loading

KoalaSat commented Jul 12, 2024

jerryfletcher21 commented Jul 12, 2024

KoalaSat commented Jul 12, 2024

jerryfletcher21 commented Jul 13, 2024

KoalaSat Jul 20, 2024 •

edited

Loading

KoalaSat commented Aug 9, 2024 •

edited

Loading

KoalaSat commented Sep 6, 2024

Use nostr as cache system #1362

Use nostr as cache system #1362

Conversation

KoalaSat commented Jul 1, 2024 • edited Loading

What does this PR do?

Federation cache

How to test

Checklist before merging

jerryfletcher21 commented Jul 12, 2024 • edited Loading

KoalaSat commented Jul 12, 2024

jerryfletcher21 commented Jul 12, 2024

KoalaSat commented Jul 12, 2024

jerryfletcher21 commented Jul 13, 2024

KoalaSat Jul 20, 2024 • edited Loading

Choose a reason for hiding this comment

KoalaSat commented Aug 9, 2024 • edited Loading

KoalaSat commented Sep 6, 2024

KoalaSat commented Jul 1, 2024 •

edited

Loading

jerryfletcher21 commented Jul 12, 2024 •

edited

Loading

KoalaSat Jul 20, 2024 •

edited

Loading

KoalaSat commented Aug 9, 2024 •

edited

Loading